United States: bibliometric analysis

Objective To compare the volume and quality of original research in primary care published by researchers from primary care in the United Kingdom against five countries with well established academic primary care. Design Bibliometric analysis. Setting United Kingdom, United States, Australia, Canada, Germany, and the Netherlands. Studies reviewed Research publications relevant to comprehensive primary care and authored by researchers from primary care, recorded in Medline and Embase, with publication dates 2001-7 inclusive. Main outcome measures Volume of published activity of generalist primary care researchers and the quality of the research output by those publishing the most using citation metrics: numbers of cited papers, proportion of cited papers, and mean citation scores. Results 82169 papers published between 2001 and 2007 in the six countries were classified as research on primary care. In a 15%pragmatic random sample of these records, 40% of research on primary care from the United Kingdom and 46% from the Netherlands was authored by researchers employed in a primary care setting or employed in academic departments of primary care. The 141 researchers with the highest volume of publications reporting research findings published between 2001 and 2007 (inclusive) authored or part authored 8.3% of the total sample of papers. For authors with the highest proportion of publications cited at least five times, the best performers came from the United States (n=5), United Kingdom (n=4), and the Netherlands (n=2). In the top 10 of authors with the highest proportions of publications achieving 20 ormore citations, six were from the United Kingdom and four from the United States. The meanHirsch index (measure of a researcher’s productivity and impact of the published work) was 14 for the Netherlands, 13 for the United Kingdom, 12 for the United States, 7 for Canada, 4 for Australia, and 3 for Germany. Conclusion This international comparison of the volume and citation rates of papers by researchers from primary care consistently placed UK researchers among the best performers internationally. INTRODUCTION The UK Research Assessment Exercise reported in late 2008. This was the sixth such national peer review evaluation of the quality of research carried out by higher education institutions, providing a quality rating for defined areas of research. This rating, converted to a multiplication factor on volume measures such as numbers of staff and research students and amount of grant expenditure onpeer review, determines the quality adjusted research funding to institutions. In 200910, the Higher Education Funding Council for England allocated £1.6bn (€1.9bn; $2.6bn) in quality adjusted research to UK institutions that had collectively attracted £3.7bn in total research grants and contracts (2007-8) and £658m in income from charitable grants (2009-10). In addition to providing an objective rationale for varying the allocation of resources to support research in institutions, the results of the Research Assessment Exercise are used for many secondary purposes, including rating the research environment for future external research awards. The Research Assessment Exercise therefore generates a cycle of quality rating, being applied prospectively both directly through the block allocation for quality adjusted research and indirectly through its “kudos” score. This may have had the effect of further concentrating research investment in the United Kingdom—the proportion of total public funding to the top 10th of researchers in the United Kingdom increased from 47% in 1980-1 to 57% in 1997-8, compared with a decline in the United States from 47% to 43%. In the 2008 Research Assessment Exercise, 75% of the total score in any given subject area was derived from the assessment of publication outputs. Lesser weighting was given to the research environment and evidenceof “esteem.”Units of assessmentwere aligned to major research themes, such as cardiovascular disease, but also recognised the methodological diversity of the more applied clinical research areas of clinical epidemiology, health services research, and primary care. The 2008 scoring system placed more emphasis on research of “international quality,” evolving from York Health Economics Consortium, University of York Department of Primary Care, University of Southampton National Primary Care Research and Development Centre, University of Manchester Department of Primary Care, Peninsula Medical School Department of Primary Care Clinical Sciences, University of Birmingham NIHR National School for Primary Care Research Correspondence to: F D R Hobbs, Primary Care Clinical Sciences Building, University of Birmingham, Edgbaston, Birmingham B15 2TT [email protected] Cite this as: BMJ 2011;342:d1028 doi:10.1136/bmj.d1028 BMJ | ONLINE FIRST | bmj.com page 1 of 10 the 2001 Research Assessment Exercise 0-5 star rating to an overall quality profile assessment of below national standard (“unclassified”), nationally recognised research (one star), internationally recognised research (two stars), internationally excellent research (three stars), and world leading research (four stars). TheResearchAssessment Exercise therefore provided a qualitative assessment (and comparison by institution), ranked by international importance, of research within the United Kingdom. The reliability of these international ratings of research importance is, however, unclear. Assessors categorised international research in the absence of any published criteria on what defines “recognised” compared with “world beating” status. In terms of crude comparisons, overall UK research tops the world league for papers per dollar expended and citations per dollar expended and is ranked fourth for papers per researcher. Furthermore, the United Kingdomdelivers 9% of theworld’s research effort for 4.5% of the world’s research expenditure (and in a high cost economy). But how does a relatively young academic discipline such as primary care compare on quality internationally? Subpanels encompassing more applied clinical research, including primary care, have historically been rated below laboratory and hospital based research inResearchAssessmentExercises. Furthermore, commentators have questioned the viability of primary care research. Despite limitations, the United Kingdommay transfer future state support for academic research from the substantially peer led Research Assessment Exercise review (high intensity, high cost) to assessments in the Research Excellence Framework, potentially informed by bibliometric indicators and expert review. Subpanels for the Research Excellence Framework may use data on citations to inform their assessments. To internationally benchmark UK research on primary care, we carried out volume and quality analyses of publications by primary care researchers from six countries with well established primary care research before the publication of the 2008 Research Assessment Exercise.